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Abstract 


Command  and  Control  for  coordinated  response  to  domestie  terrorist  attaek  will  require  the 
ability  of  federal,  state,  and  loeal  agencies  to  maintain  awareness  of  the  status,  capabilities, 
requirements,  response  plans,  and  C2  procedures,  etc.  of  the  other  eollaborating  organizations. 
While  progress  is  being  made  in  improving  information  sharing,  the  TOPOFF  exereises  have 
demonstrated  that  organizations  still  laek  any  substantial  ability  to  eoordinate  responses  to  large 
scale  events  that  involve  dozens  of  local,  state,  and  federal  organizations.  In  this  paper  we 
deseribe  progress  made  in  the  development  of  new  information  aeeess  services  that  provide  for 
improved  situation  awareness.  We  have  strived  to  develop  a  solution  that  enables  User  Defined 
Operational  Pieture  (UDOP)  funetionality  while  respeeting  the  unique  information  management 
practices  of  the  collaborating  Homeland  Security  organizations.  Our  system  eoneept,  Vista, 
employs  an  adaptive  maehine  learning  paradigm  that  supports  a  new  form  of  eontext-sensitive 
information  aeeess,  monitoring,  and  alerting  that  fills  substantial  gaps  in  existing  Crisis 
Information  Management  System  teehnologies.  Experimental  results  demonstrate  very 
substantial  improvements  in  information  aeeess  effieieney  and  provide  strong  evidenee  for  the 
feasibility  of  the  overall  concept. 

1  Introduction 

Homeland  Seeurity  (HLS)  is  rightly  a  high  priority  at  all  levels  of  the  government,  and  huge 
investments  are  being  made  to  train  and  provide  basic  resources  to  our  country’s  first  responders. 
We  are  also  seeing  dramatic  changes  in  our  intelligence  ageneies  in  order  to  improve  information 
sharing  aeross  organizational  boundaries  (e.g.,  the  formation  of  the  Terrorist  Threat  Integration 
Center  (TTIC)  and  the  ereation  of  information  portals  sueh  as  the  Northwest  Warning,  Alert  & 
Response  Network  (Northwest  WARN)).  Further,  the  DoD  is  making  available  critical  assets 
through  organizations  sueh  as  the  Army’s  Guardian  Brigade  which  will  provide  military 
assistanee  to  eivilian  law  enforeement  offieials  in  emergeney  situations  involving  biologieal  or 
ehemical  weapons. 

Unfortunately,  there  are  three  very  signifieant  faetors  that  eomplieate  the  situation.  First,  and 
most  obvious,  are  the  range  of  institutional  and  soeial  barriers  that  have  inhibited  the  exehange  of 
information  amongst  federal,  state,  and  loeal  agencies.  Intelligence  sharing  and  eoordination  can 
be  expeeted  to  be  problematieal  due  to  the  general  laek  of  a  unified  and  hierarehieal  eommand  in 
situations  involving  dozens  of  collaborating  governmental  and  non-govemmental  organizations 
which  all  operate  with  high  degrees  of  autonomy  and  their  own  missions/goals  (partieularly  in 
so-called  “home  rule”  states). 


The  second  complication  is  a  general  lack  of  system  and  semantic  interoperability  amongst 
HLS  organizations.  While  modern  information  technology  obviously  has  the  potential  to 
provide  great  improvements  in  situational  awareness  and  coordination,  in  reality  high  technology 
is  just  now  making  substantial  inroads  into  Emergency  Management  (EM).  The  DHS  report 
following  TOPOEE  2  highlighted  this  problem  and  in  particular  discussed  the  heavy  reliance  on 
the  most  basic  forms  of  communication  (e.g.,  face  to  face  meetings,  phone,  fax,  and  radio).  This 
problem  together  with  the  lack  of  a  shared  vocabulary  obviously  limited  the  ability  of  these 
heterogeneous  EM  organizations  to  produce  a  common  operational  picture. 

Efforts  are  being  made  to  improve  the  situation.  Eor  example,  the  Homeland  Security 
Presidential  Directive  (HSPD)-5  directs  the  Department  of  Homeland  Security  (DHS)  to  develop 
and  administer  a  National  Incident  Management  System  (NIMS)  (see  NIMS  (2005)): 

“This  system  will  provide  a  consistent  nationwide  approach  for  Federal,  State,  and  local 
governments  to  work  effectively  and  efficiently  together  to  prepare  for,  respond  to,  and  recover 
from  domestic  incidents,  regardless  of  cause,  size,  or  complexity.  To  provide  for 
interoperability  and  compatibility  among  Federal,  State,  and  local  capabilities,  the  NIMS  will 
include  a  core  set  of  concepts,  principles,  terminology,  and  technologies  covering  the  incident 
command  system;  multiagency  coordination  systems;  unified  command;  training;  identification 
and  management  of  resources  (including  systems  for  classifying  types  of  resources); 
qualifications  and  certification;  and  the  collection,  tracking,  and  reporting  of  incident 
information  and  incident  resources.  ” 

Of  course  the  practical  benefits  of  these  efforts  may  take  substantial  time  to  manifest.  This 
leads  to  the  third  significant  complication  impacting  the  coordination  of  HES  organizations 
which  is  a  simple  lack  of  practice.  The  TOPOEE  exercises  are  some  of  the  few  significant 
exercises  involving  a  coordinated  response  to  a  large  scale  terrorist  threat.  The  infrequency  of 
these  large  scale  exercises  is  not  at  all  surprising  given  their  high  cost  ($16  million  in  the  case  of 
TOPOEE  2).  It  is  important  to  note  that  smaller  scale  exercises  held  at  the  state  and  local  levels 
prove  very  effective  in  improving  the  ability  of  EM  organizations  to  coordinate  in  crisis 
situations,  but  these  very  seldomly  involve  Eederal  agencies  or  National  Guard  participants.  The 
resulting  lack  of  familiarity,  shared  goals,  and  common  understandings  caused  very  substantial 
problems  during  the  TOPOEE  2  exercise  that  will  likely  only  be  amplified  in  a  real  HES  crisis  - 
causing  problems  in  coordination  as  well  as  a  simple  lack  of  awareness  of  the  full  range  of 
resources  that  could  be  brought  to  bear. 

In  this  paper  we  describe  progress  made  developing  new  technologies  that  can  help 
overcome  the  above  challenges  and  complement  recent  progress  in  the  development  and  use  of 
Crisis  Information  Management  System  technologies.  Our  overall  goal  is  to  improve  situation 
awareness  and  thereby  the  ability  to  coordinate  crisis  response.  More  specifically,  we  have 
established  a  phased  approach  which  first  focuses  on  improving  existing  workflow  processes 
used  by  EM  organizations  to  access  and  monitor  key  information  across  institutional  boundaries. 
Over  the  longer  term  we  will  develop  methods  for  the  efficient  establishment  and  maintenance  of 
a  new  level  of  semantic  interoperability  that  will  in  turn  allow  a  much  higher  level  of  automation 
in  delivering  the  military's  desired  User  Defined  Operational  Picture  (UDOP)  functionality  that 
can  provide  improved  awareness  of  unfolding  events,  the  disposition  of  available  resources,  etc. 

The  remainder  of  this  paper  is  organized  as  follows.  Eirst  we  review  some  of  the  key  lessons 
we  learned  in  our  work  with  Washington  State  Emergency  Management  organizations  who 
played  key  roles  in  the  TOPOEE  2  exercise.  We  will  then  describe  the  current  state  of  the  art  in 


Crisis  Information  Management  Systems  and  pinpoint  the  teehnology  gaps  we  seek  to  fill.  We 
will  then  deseribe  the  progress  we  have  made  in  developing  information  aeeess  and  monitoring 
tools  that  ean  streamline  existing  workflows  and  inelude  some  early  experimental  results.  We 
will  eonelude  with  a  deseription  of  our  future  efforts. 

2  The  Practicalities  of  All  Hazards  Emergency  Management 

In  order  to  understand  how  the  work  deseribed  herein  ean  support  HLS  operations,  it  is  first 
important  to  plaee  HLS  in  the  eontext  of  eommon  EM  praetiee.  There  are  in  faet  several  key 
issues  that  must  be  eonsidered  in  order  to  suoeessfully  field  teehnologies  that  ean  support  HLS. 
First,  the  proeedures  followed  in  HLS  operations  are  not  separable  from  general  EM  praetiee. 
Instead  Emergeney  Operations  Centers  (EOCs)  operate  under  so-ealled  “all-hazards”  emergeney 
response  strategie  plans  whieh  stipulate  under  what  eonditions  the  EOC  is  aetivated,  who 
ultimately  direets  the  response,  and  how  outside  resourees  (e.g.,  military  personnel)  are  aequired 
and  exploited.  More  importantly  these  plans  speeify  that  basie  polieies,  proeedures,  and  ehain  of 
eommand  are  the  same  whether  an  emergeney  is  due  to  an  earthquake,  a  disease  outbreak,  or  a 
terrorist  attaek. 

Seeond,  eonsider  that  in  “home-rule”  states  sueh  as  Washington  State,  a  eounty  EOC  must 
eoordinate  dozens  of  largely  autonomous  government  and  non-government  organizations  -  eaeh 
with  their  own  eonstitueneies,  resourees,  and  needs.  Even  in  non-home -rule  states,  eoordination 
with  autonomous  non-governmental  organizations  poses  the  same  ehallenges  to  eommand  and 
eontrol  sinee  information  sharing  and  general  eooperation  is  voluntary.  For  these  reasons 
eommunieations  are  ad  hoe  relative  to  that  witnessed  in  military  organizations. 

Finally,  we  must  eonsider  that  emergeney  response  eoordination,  while  the  point  of  emphasis 
for  the  DHS  and  the  press,  represents  a  very  small  portion  of  the  responsibilities  of  EM 
organizations.  The  full  time  staff  of  these  organizations  spends  the  vast  majority  of  their  time  in 
tasks  related  to  emergeney  preparation  and  reeovery.  Further,  sinee  EOC’s  are  seldom  fully 
aetivated  and  training  exereises  are  infrequent,  one  ean  expeet  personnel  to  faee  signifieant 
ehallenges  in  exploiting  teehnologieal  tools  when  a  erisis  does  oeeur.  These  faets  highlight  how 
eritieal  it  is  that  tools  be  easy  to  use.  Perhaps  even  more  importantly,  teehnologies  that  solely 
support  the  least  likely  of  events  (e.g.,  large  seale  HLS  events)  are  unlikely  to  be  adopted  by  EM 
organizations. 

2.1  The  Joint  Operations  Center  Concept 

Another  important  eoneept  is  that  of  the  Joint  Operations  Center  (JOC),  whieh  aeeording  to 
state  and  loeal  EM  strategie  plans,  will  be  established  by  the  loeal  FBI  offiee  in  the  event  of  a 
signifieant  HLS  event.  The  purpose  of  the  JOC  is  to  provide  a  loeation  where  trusted 
representatives  of  Federal,  State,  and  Loeal  ageneies  ean  meet  faee  to  faee  to  resolve  eritieal 
issues  and  support  a  well  eoordinated  response. 

From  the  perspeetive  of  Seattle  and  King  County  EM,  the  JOC  serves  little  purpose  sinee  the 
liaisons  sent  to  the  JOC  will  most  often  not  be  key  deeision  makers,  nor  does  it  replaee  anyone's 
exeeutive,  politieal,  eommand,  or  legal  authority.  Further,  as  we  will  diseuss  in  the  next  seetion, 
it  appears  that  modern  eommunieations  teehnology  provides  more  than  sufficient  support  for 
executive  decision  makers  to  coordinate  from  a  distance.  This  finding  is  in  line  with  the 
hypotheses  presented  by  Mayk  and  Klose  (2004)  with  regards  to  military  C2. 


The  JOC  in  TOPOFF  2  was  the  souree  of  several  problems  according  to  Seattle’s  then 
Director  of  Emergency  Management,  Jim  Mullen: 

“During  the  exercise,  an  unauthorized  evacuation  was  in  the  early  stages  of  planning  before 
our  EOC — the  command  and  control  center  for  the  event — even  heard  about  it.  Fortunately,  we 
were  able  to  stop  it,  but  the  lesson  was  that  if  you  have  a  place  where  people  with  initiative  but 
without  authority  are  congregated,  they  will  sometimes  do  more  harm  than  good” 

3  State  of  the  Art  in  Crisis  Information  Management  Tools 

In  order  to  support  the  activities  of  the  EOC,  each  is  equipped  with  a  wide  range  of 
communications  equipment  including  EEMA  National  Warning  System  (NAWAS)  terminals. 
Emergency  Alert  System  (EAS)  radio.  Computer  Aided  Dispatch  (CAD)  systems,  lines  to  the 
ATE  and  EBI,  and  digital  and  analog  phone  systems.  These  systems  provide  the  baseline  means 
for  maintaining  situation  awareness  and  for  supporting  coordination  with  outside  organizations. 

EOC’s  are  also  undergoing  some  dramatic  changes  with  regards  to  technology  support.  In 
recent  years,  teleconferencing  systems  have  been  installed,  and  EOC’s  are  making  a  gradual  shift 
from  a  reliance  on  grease  pencils  and  laminated  boards/maps  to  modern  crisis  management  tools. 
Eor  instance,  the  three  Washington  State  EM  organizations  examined  in  our  research  use  a  tool 
called  WebEOC  to  support  the  management  and  online  dissemination  of  crisis  information.  This 
tool  offers  a  variety  of  so-called  “status  boards”  that  are  used  to  post  and  track  the  happenings 
within  the  collaborating  organizations.  WebEOC  and  similar  tools  such  as  E  Team  at  their  core 
are  simple  information  sharing  tools  -  essentially  replicating  physical  bulletin  boards  that  have 
been  used  in  EOC’s  for  many  years  and  making  them  accessible  over  computer  networks.  Other 
features  allow  for  incident  replay  for  training  purposes,  alert  dissemination,  resource  tracking, 
and  limited  incident/resource  mapping. 

Despite  the  promise  of  these  tools,  there  are  a  number  of  emerging  difficulties.  Eirst  of  all, 
according  to  a  recent  Department  of  Justice  study  (DOJ,  2002)  there  are  approximately  a  dozen 
different  Crisis  Information  Management  Systems  (CIMS)  in  use  by  HES  relevant  organizations 
throughout  the  country,  and  the  developers  have  come  late  to  the  concept  of  interoperability. 
Progress  is  being  made  to  develop  an  Emergency  Management  XME  schema  to  improve 
interoperability,  but  as  yet  this  schema  is  limited  to  the  so-called  Common  Alerting  Protocol 
(CAP)  which  was  recently  released  by  a  consortium  of  CIMS  developers  (CAP,  2004)  and 
provides  a  template  for  emergency  alerts.  Because  of  its  limited  nature,  CAP  does  little  to 
support  advanced  incident  coordination  applications. 

Another  challenge  facing  users  of  existing  tools  relates  to  the  supported  information  access 
paradigm.  While  CIMS  tools  allow  for  inter-organizational  publishing  and  sharing  of 
information  across  computer  networks,  information  publishers  and  consumers  are  often  out  of 
synch.  That  is,  the  information  “pushed”  to  consumers  is  often  not  relevant  to  their  ongoing 
tasks  and  the  information  “pull”  mechanisms  may  require  much  manual  searching  and 
monitoring.  In  fact,  the  EM  organizations  we  examined  in  this  research  had  to  assign  teams  of 
personnel  to  continually  monitor  various  WebEOC  status  boards  and  external  Web  data  sources 
(e.g.,  news  coverage,  road  closures,  etc.)  during  activations  in  order  to  ensure  key  information  is 
delivered  to  those  that  need  it.  Of  course  the  challenges  of  timely  information  access  and  fusions 
will  become  even  greater  as  the  variety  of  information  and  information  providers  increases  to 
allow  monitoring  of  available  hospital  beds,  power  outage  areas,  the  course  of  chemical  plumes, 
etc. 


4  Opportunities  to  Improve  HLS  Emergency  Response 

TOPOFF  2  and  the  experiences  of  state  and  local  EM  organizations  in  more  conventional 
operations  suggest  that  there  remain  many  opportunities  to  improve  HLS  situation  awareness  and 
response  coordination  capabilities.  Obviously,  communications  and  coordination  capabilities 
amongst  HLS-relevant  DoD,  non-DoD,  Federal,  state,  and  local  organizations  is  not  what  it 
should  be.  In  the  event  of  significant  domestic  terrorism,  response  time  is  critical  and  missteps 
can  have  extremely  high  costs  in  human  lives.  Therefore,  new  technologies  are  required  that  can 
dynamicallyexploit  the  output  of  new  information  providers  to  offer  both  vastly  improved 
information/situation  awareness  and  the  ability  to  coordinate  crisis  response.  With  that  said,  we 
recognize  that  new  technologies  face  a  number  of  challenges  in  achieving  adoption  by  EM 
organizations  -  most  substantially  the  need  to  fit  with  existing  workflows  and  systems  and 
provide  immediate  benefit  in  day  to  day  operations,  as  well  as  maintain  low  training 
requirements.  With  this  in  mind  we  are  pursuing  a  phased  R&D  approach  where  early  software 
deployments  can  inform  us  as  to  the  remaining  areas  of  need. 

We  currently  are  pursing  three  key  opportunities,  with  the  first  being  our  initial  focus. 

1.  Intelligent  context-sensitive  information  access,  monitoring,  and  alerting  across 
organizational  boundaries. 

Seizing  this  technological  opportunity  can  provide  context  relevant  information  access  so 
that  EOC  personnel,  military  commanders,  intelligence  analysts,  etc.,  can  quickly  identify 
and  track  the  information  that  they  know  they  need  to  make  time-critical  decisions,  as  well  as 
provide  alerts  with  decision  impacting  information  they  did  not  even  recognize  they  needed. 

This  new  information  access  capability  exploits  a  model  of  the  current  operational  context  to 
provide  automated  Web  site  and  status  panel  monitoring,  federated  search  across  cooperating 
organizations,  and  to  provide  users  with  an  ongoing  awareness  of  the  information  and 
knowledge  being  generated  and  exploited  across  HLS  organizations  that  fills  local 
information  needs. 

2.  The  efficient  establishment  and  maintenance  of  semantic  interoperability. 

Over  the  longer  term  we  are  seeking  to  develop  techniques  that  can  build  upon  the  progress 
made  by  US  Army  C2  CERDEC  in  the  development  of  C2XML  (see  Mayk  and  Klose, 
(2004)),  the  EM-XML  consortium  in  the  development  of  CAP  (CAP,  2004),  and  the  DHS’s 
recent  efforts  to  develop  the  National  Response  Plan  (NRP)  and  the  National  Incident 
Management  System  (NIMS)  in  order  to  produce  a  mediating  ontology  that  will  be  necessary 
to  achieve  interoperability  across  the  legacy  and  future  systems  operated  by  HLS  relevant 
Eederal,  DoD,  and  local  EM  organizations. 

This  type  of  rigorous  information  model  has  much  to  offer,  including  the  potential  of 
allowing  state  and  local  EM  organizations  to  bring  to  bear  the  valuable  collaborative 
planning  systems  (e.g.,  CAPES/MC2)  employed  by  the  US  military.  It  will  also  open  the 
possibility  of  lower-cost,  multi-agency,  computer  based  training  exercises.  This  type  of  asset 
sharing  represents  a  novel  and  valuable  direction  in  HLS. 

3.  A  vastly  improved  capability  to  provide  awareness  of  unfolding  events,  the  disposition 
of  available  resources,  etc,,  through  a  User  Defined  Operational  Picture, 


The  aforementioned  ontologieal  foundation  may  provide  a  suffieient  basis  for  the  automated 
processing  and  formalization  of  C2  information  such  as  significant  event  messages,  EM 
tasking  orders,  Operation  Orders  (OPORDs),  etc.,  and  to  support  information  system 
integration.  These  new  capabilities  will  allow  us  to  move  beyond  the  current  generation  of 
manually  updated  electronic  bulletin  boards,  and  on  to  the  production  of  a  common  operating 
picture  that  can  be  tailored  to  the  user  organization. 

Not  only  will  such  displays  offer  the  EM  community  a  substantial  improvement  in 
situational  awareness,  it  will  prove  an  essential  capability  in  achieving  effective 
coordination  with  organizations  outside  the  local  civilian  EM  community. 

5  The  Vista  Concept 

Our  initial  Vista  prototype  focused  on  the  task  of  extending  and  evaluating  an  existing 
context-sensitive  information  retrieval  and  monitoring  technology  (see  Aware  (2005))  in  order  to 
improve  the  ability  of  users  to  locate  and  maintain  awareness  of  operation  critical  information. 

In  particular  we  have  prototyped  and  tested  a  new  search  scheme  that  mitigates  the  inevitable 
problem  of  a  users’  query  terminology  not  matching  with  that  employed  by  the  authors  of  the 
desired  information.  This  problem  crops  up  in  all  information  retrieval  tasks  simply  because  it  is 
difficult  for  the  information  consumer  to  fully  anticipate  the  terminology  the  information 
producer  has  utilized,  and  just  as  commonly  users’  queries  can  unintentionally  match 
information  that  is  irrelevant  to  their  task.  These  challenges  are  particularly  obvious  in  situations 
where  information  is  being  accessed  across  organizational  boundaries,  such  as  in  the  case  of 
EM/HLS  operations. 


Figure  1.  Vista  Prototype  Overview 


The  scheme  we  have  implemented  has  the  following  primary  components. 

1 .  Context  Modeling.  Our  search  technology  has,  at  its  core,  the  ability  to  derive  a  model 
of  the  user’s  information  need  through  an  analysis  of  context  information.  This  context 


typically  is  composed  of  a  set  of  rated  (relevant  /  non-relevant)  doeuments,  but  ean  also 
be  derived  through  much  less  explicit  means  by  monitoring  the  user’s  information 
management  operations.  In  the  eontext  of  this  project,  this  may  involve  the  status  of 
personal  or  EOC  aetivity  logs  complete  with  hyperlinks  to  information  used  to  form 
decisions. 

2.  Multi-search,  Leveraging  the  user’s  operational  context,  our  teehnology  is  able  to 
produce  multiple  refinements  of  user  queries  in  order  to  scan  a  topic  area.  In  the  eurrent 
effort  we  have  made  a  number  of  improvements  to  this  teehnology  that  make  it  feasible 
for  Vista  to  autonomously  seareh  for  information  relevant  to  the  user’s  expressed  (via  the 
operation  context)  information  requirements.  This  advanee  required  new  techniques  that 
are  able  to  distinguish  key  terminology  (what  we  call  "query  anchors")  from  other 
terminology  that  is  only  vaguely  relevant  (see  Figure  2). 


Figure  2.  Vista's  Mission  Context  Model.  Vista  processes  the  documents  contained  in  the  user’s 
Mission  Context  (i.e.,  the  document  collection)  and  extracts  out  the  key  terminology  defining  the 
user’s  requirements.  This  list  of  terminology  is  continually  refined  hy  the  user’s  rating  of  documents 
and  through  Vista’s  concept  switching  functionaUty.  Note  that  terms  have  a  user  modifiable  rating,  a 
Traction  score  (T),  and  a  system  generated  score  (S). 

3.  Traction,  One  of  our  biggest  sueeesses  in  this  projeet  has  been  the  prototyping  of  a 
method  that  is  able  to  measure  the  “traetion”  of  query  terms  in  delivering  information  that 
the  user  requires.  Based  on  binary  relevanee  feedbaek,  Vista  is  able  to  quiekly  identify 
whieh  query  anehors  are  most  relevant  to  the  user’s  eurrent  tasks,  and  most  effective  at 
identifying  the  required  information  (see  Figure  2). 

4.  Enhanced  concept  switching.  The  other  major  advanee  we  have  made  in  our 
prototyping  efforts  is  an  extension  of  so-called  “concept  switehing”  (see  Sehatz  (2002)). 
Concept  switehing  is  a  mechanism  for  dealing  with  situations  where  the  information 
seeker  may  not  know  the  appropriate  query  terminology  needed  to  aeeess  information 


developed  in  another  community/organization.  Coneept  switehing  starts  with  the  one 
core  concept  in  the  user’s  context,  generates  a  set  of  related  concepts  in  that  space,  and 
then  uses  this  broader  set  to  locate  the  most  likely  matching  concept  in  the  target 
community.  In  our  prototype  implementation  of  an  enhancement  of  this  concept,  we 
allow  the  automatic  acquisition  of  terminology  by  a  user's  (or  community's)  Vista  system 
from  the  terminology  (ontology)  employed  in  another  community’s  Vista  system,  based 
on  the  presence  of  shared  interests.  We  then  employ  the  aforementioned  traction 
heuristics  to  validate  that  the  terminology  acquired  is  indeed  relevant  to  the  user’s  task. 

5.  Context-based  information  evaluation  and  filtering.  Finally,  Vista  is  able  to  evaluate 
the  relative  quality  of  discovered  information  based,  not  only  on  a  limited  query,  but  on 
the  entirety  of  the  search  context.  This  more  comprehensive  model  leads  to  greater 
accuracy  in  information  ranking/filtering  and  even  enables  the  system  to  measure  the 
novelty  of  new  information  (which  may  be  indicative  of  a  significant  turn  of  events). 

6  Preliminary  Experiments 

In  order  to  test  the  efficacy  of  our  newly  prototyped  technologies,  we  designed  two 
experiments.  In  the  first  experiment,  we  used  human  test  subjects  to  compare  the  efficacy  of 
search  via  traditional  search  tools  versus  the  Vista  context-aware  search  methods.  In  the  second 
we  tested  the  viability  of  our  enhanced  concept  switching  method. 

In  these  preliminary  experiments  we  relied  on  a  relatively  simple  search  task  specification  in 
order  to  evaluate  the  effectiveness  of  our  techniques.  In  this  test  scenario  we  have  two 
communities  of  users,  the  CIA  and  the  FBI,  who  are  investigating  the  2001  Anthrax  letter 
attacks.  Each  community  has  its  own  unique  search  objectives  (the  CIA  is  investigating  links  to 
international  terrorist  organizations  and  foreign  states  like  Iraq,  while  the  FBI  is  examining 
domestic  suspects)  and  terminology,  but  clearly  there  is  a  substantial  overlap  in  information  that 
would  be  deemed  relevant  to  their  respective  investigation. 

6.1  Testing  Context  Based  Search 

For  this  study  we  recruited  ten  subjects  from  within  our  company,  whose  occupations  ranged 
from  technical  writer  to  software  engineer.  One  half  of  this  group  was  randomly  selected  to  act 
as  a  Control  group,  and  were  asked  to  produce  a  total  of  four  queries  to  the  Google  search  engine 
and  review  at  most  ten  pages  per  query.  We  did  not  prevent  the  users  from  examining  multiple 
pages  of  search  results,  but  each  page  of  ten  results  counted  toward  their  four  query  limit. 
Subjects  were  then  asked  to  rank  the  results  as  Unrelated,  Related,  or  Highly  Related. 

The  other  half  of  our  study  subjects  acted  as  the  Experimental  group.  Each  person  in  this 
group  was  only  allowed  two  queries,  and  was  given  the  same  instructions,  but  was  also  asked  to 
provide  our  search  tool  with  the  pages  they  found  highly  related  to  the  topic.  Our  search  tool 
used  the  resulting  set  of  texts  as  a  context  for  one  additional,  automatically  constructed  query 
which  was  used  to  fetch  fifty  results  and  rank  them  -  reserving  only  the  top  20. 

The  results  of  this  experiment  were  extremely  encouraging.  We  found  that  our  search  system 
was  able  to  make  very  good  use  of  the  small  efforts  taken  by  the  Experimental  group  to  provide 
feedback.  In  fact,  it  was  able  to  locate  results  that  were  considerably  better  than  those  located 
using  the  queries  constructed  by  hand  by  the  control  group  after  they  had  analyzed  two  pages  of 
search  results.  In  particular,  while  the  Control  group  saw  a  56%  increase  in  the  number  of  high 
quality  pages  resulting  from  their  second  two  queries  (in  comparison  to  the  first  two),  the 


Experimental  group  saw  a  jump  of  127%  (see  Table  1).  The  results  were  equally  impressive 
when  eonsidering  doeuments  rated  as  either  Related  or  Highly  Related  (see  Table  2). 

Also  note  that  we  separated  Test  Subjeet  10  from  the  rest  of  the  Experimental  group,  beeause 
he  had  some  amount  of  training  in  the  use  of  the  software  and  was  therefore  better  able  to 
provide  texts  that  would  direet  the  system.  It  is  interesting  that  Test  Subjeet  10  did  not  do 
substantially  better  than  those  subjects  who  had  not  been  given  any  instruction  in  how  to  best 
train  the  system. 


Control  Group 

Highly  Related 
(1*‘  2  queries) 

Highly  Related 
(2nd  2  queries) 

%  change 

1st  half  to  2nd  half 

Subject  1 

1 

3 

200 

Subject  2 

5 

9 

80 

Subject  3 

1 

3 

200 

Subject  4 

2 

0 

-too 

Subject  5 

3 

0 

-too 

Average 

2.4 

3 

56 

Experimental  Group 

Subject  6 

4 

10 

150 

Subject  7 

2 

6 

200 

Subject  8 

4 

6 

50 

Subject  9 

5 

9 

80 

Average 

3.75 

7.75 

120 

Subject  10 

7 

16 

128.6 

Table  1.  Tbe  Search  for  Highly  Related  Pages 


Control  Group 

Total  Related 
(1*‘  2  queries) 

Total  Related 
(2"“  2  queries) 

%  change 

1st  half  to  2nd  half 

Subject  1 

12 

5 

-58.3 

Subject  2 

9 

14 

55.6 

Subject  3 

4 

7 

75.0 

Subject  4 

4 

1 

-75.0 

Subject  5 

5 

5 

0.0 

Average 

6.8 

6.4 

-0.6 

Experimental  Group 

Subject  6 

9 

14 

55.6 

Subject  7 

5 

18 

260.0 

Subject  8 

15 

12 

-20.0 

Subject  9 

9 

14 

55.6 

Average 

9.5 

14.5 

87.8 

Subject  10 

11 

20 

81.8 

Table  2.  The  Search  for  Related  Pages  (including  Highly  Related) 


6.2  Testing  Enhanced  Concept  Switching 

In  this  preliminary  experiment  we  utilized  a  simpler  evaluation  seheme  for  expedience.  In 
this  experimental  set  up  we  constructed  two  search  profiles  -  one  representing  the  FBI  and  one 
the  CIA.  In  each  case  we  had  previously  conducted  intensive  manual  searches  to  identify  pages 
relevant  to  the  search  profiles  (as  well  as  a  set  of  off-topic  pages).  We  then  began  a  batch  search 
process  that  proceeded  as  follows: 

1.  An  initial  query  was  executed  for  the  CIA  profile  (e.g.,  ‘anthrax  terrorist  iraq’).  Any 
documents  that  were  returned  by  the  search  engine  were  compared  to  the  associated  list 
of  relevant  and  off-topic  documents  and  Vista  was  provided  with  the  appropriate 
feedback  as  if  the  human  user  were  involved. 

2.  An  initial  query  was  executed  for  the  FBI  profile  (e.g.,  ‘"anthrax  letters"  "ames  strain"’). 
Feedback  was  provided  as  above. 

3.  If  any  document  was  located  that  was  judged  as  relevant  to  both  the  FBI  and  the  CIA, 
Vista  allowed  terminology  to  cross  from  one  profile  to  the  other.  In  particular,  the  query 
terms  that  matched  the  particular  results  were  tentatively  assumed  to  be  relevant  to  the 
other  search  profile.  Following  the  crossover,  traction  was  employed  to  validate  the 
utility  of  the  term  in  the  search  profile  -  degrading  it  if  the  term  failed  to  produce  more 
relevant  results. 

4.  A  series  of  interleaved  queries  (CIA,  FBI,  CIA,  etc.)  was  then  automatically  produced, 
each  followed  by  appropriate  feedback  and  terminology  crossover  operations. 

5.  Finally,  the  number  of  relevant  documents  retrieved  was  tabulated 

In  this  experiment  we  allowed  a  total  of  40  searches  per  search  profile  (simulating  multiple 
users  in  each  organization.) 

The  results  from  this  experiment  were  surprisingly  good.  The  test  runs  employing  our 
enhanced  concept-switching  returned  an  average  of  45%  of  the  known  good  FBI  documents  (as 
compared  to  only  32%  when  sharing  was  not  employed),  and  an  average  of  35%  of  the  known 
good  CIA  documents  (as  compared  to  31%  when  sharing  was  not  employed).  These  are 
substantial  improvements  in  recall  for  a  technique  only  a  few  months  under  development. 

Perhaps  even  more  interesting  is  the  quality  of  the  terms  that  participated  in  the  terminology 
cross-over  (e.g.,  “A1  Qaeda  operative,”  “anthrax  attacks”  “Ames  strain  of  Anthrax”).  These 
terms  are  surprisingly  good  given  the  remarkably  simple  statistical  method  we  use  to  identify 
these  multi-word  terms.  This  success  is  a  strong  indicator  that  our  traction  scheme  is  very 
successful,  since  terms  extracted  from  documents  are  continuously  reevaluated  based  on  their 
ability  to  produce  good  results  -  the  more  good  results  a  term  generates,  the  more  likely  it  is  to 
be  shared. 

7  Related  Work 

7.1  Crisis  Management 

A  number  of  companies  are  now  offering  crisis  management  software  which  supports  the 
critical  tasks  of  EM.  These  tools  support  information  sharing  in  applications  such  as  homeland 
security,  business  continuity,  disaster  preparedness  and  recovery,  event  management,  and 
training  exercises  (see  a  survey  by  the  Department  of  Justice  (DOJ,  2002).  There  also  exist  a 
number  of  examples  of  crisis  decision  support  concepts  under  development  within  the  research 


community.  One  example  is  the  EMERRS  (Emergeney  Regional  Response  System)  applieation 
(Pohl,  2001)  whieh  seeks  to  promote  effeetive  emergency  response  planning  in  urban  response 
units.  EMERRS  is  intended  to  integrate  data  from  disparate  sourees  into  a  single  eoherent  view. 
But  while  the  investigators  intend  for  this  system  to  provide  a  diseiplined  deeision-making 
environment,  it  is  eurrently  limited  to  data  integration  eapabilities  and  does  not  support  UDOP 
funetionality. 

7.2  Ontology  Alignment  and  Information  Access 

Tools  for  merging  or  aligning  ontologies  help  users  find  similarities  and  differenees  between 
souree  ontologies.  These  tools  either  identify  potential  eorrespondenees  automatieally,  or 
provide  the  environment  for  the  users  to  find  and  define  these  eorrespondenees,  or  both. 
Merging/Aligning  tools  are  often  extensions  of  development  tools.  Prominent  tools  inelude: 
PROMPT  (Noy  and  Musen,  2000),  ONION  (Mitra,  et  ah,  2000),  Chimaera  (MeGuinness  et  ah, 
2000),  and  ECA-Merge  (Stumme  and  Madehe,  2001).  See  (Noy  and  Musen,  2002)  for  a  survey. 
While  the  approaehes  employed  by  these  different  systems  vary  substantially,  one  thing  that  all 
automated  methods  have  in  common  is  that  they  rely  on  the  strueture  of  the  ontologies  to  drive 
the  alignment  proeess.  This  differs  substantially  from  the  approaeh  used  by  Vista,  where  we 
seek  to  exploit  the  text  matching  different  information  extraetion  ontologies  to  aid  in  proper 
alignment  and  merging. 

Sehatz  (2002)  deseribes  their  Interspaee  Prototype,  an  analysis  environment  supporting 
semantie  indexing  on  eommunity  repositories.  The  most  signifieant  aspeet  of  this  work  is  their 
approaeh  to  eoneept  extraetion  and  eoneept  spaces,  whieh  use  semantie  indexing  to  faeilitate 
concept  navigation.  Of  particular  interest  is  their  notion  of  “coneept-switehing”  as  a  meehanism 
for  dealing  with  situations  where  the  information  seeker  may  not  know  the  appropriate  query 
terminology  to  use  in  another  eommunity’ s  knowledge  spaee.  Coneept  switehing  starts  with  one 
eoneept  in  one  eommunity  space,  identifies  a  set  of  related  eoncepts  in  that  spaee,  then  loeates 
similar  related  sets  in  another  spaee.  Simulated  annealing  (or  the  like)  ean  then  be  used  to 
identify  the  elosest  mateh  to  the  original  eoneept. 

8  Conclusions  and  Future  Work 

We  have  established  the  teehnieal  feasibility  of  our  Vista  eoneept  in  multiple  ways.  In  this 
effort,  we  have  worked  interaetively  with  deeision  makers  in  three  Washington  State  EOC’s  to 
establish  a  good  fit  between  teehnology  and  need  -  with  a  focus  on  how  best  to  faeilitate 
information  sharing  and  eoordination.  Seeond,  our  experimental  results  (although  limited) 
provide  evidenee  that  our  techniques  for  eontext-aware  information  aceess,  monitoring,  and 
alerting  will  provide  immediate  benefits  to  the  EM/HES  eommunity,  thus  promoting  user 
adoption. 

Our  Vista  projeet  is  a  work  in  progress.  Most  reeently,  we  eommeneed  another  round  of 
interviews  and  workflow  studies  with  Washington  State  EM  organizations.  Based  on  the 
aeeumulated  information,  we  have  developed  a  preliminary  design  for  the  first  version  of  Vista 
that  will  be  deployed  in  an  EOC.  This  refined  eoneept  will  enhanee  the  eommon  operating 
pieture  for  Emergeney  Management  workers  during  the  emergeney  response  phase  by: 

•  Automating  manual  searehes  for  eritieal  information  on  agency  internet  and  intranet  websites 

•  Providing  a  eombined  and  organized  view  of  the  ineident  and  its  various  aspeets  through 

intelligent  displays  and  inferenee 


Finding  underreported  pieees  of  relevant  data  and  routing  that  information  to  the  appropriate 
decision-maker 


•  Autonomously  monitoring  external  and  internal  electronic  data  sources  and  integrating 
changes  into  the  common  operating  picture 

During  the  other  phases  of  Emergency  Management,  Vista  will  continue  to  monitor 
information  sources  and  alert  users  when  triggered  by  specified  events. 

The  potential  benefits  of  Vista's  improved  situational  awareness  is  achieved  significant  buy- 
in  from  Washington  State  EM/HLS  organizations  and  they  are  actively  involved  in  our 
requirements  analysis  and  Beta  software  testing. 
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